Method and apparatus for motion vector estimation

ABSTRACT

A method to enhance motion estimation is provided. The method includes providing a motion estimator, obtaining at least two candidate motion vectors from the motion estimator and applying an error function having a penalty that depends on the position and size of the candidate motion vectors. A device for recursive motion vector estimation having enhanced convergence is also provided. The device includes a vector generator and a best vector selector. The best vector selector comprises means to evaluate candidate motion vectors by applying an error function having at least a penalty that depends on the position and size of the candidate motion vectors.

BACKGROUND OF THE INVENTION

[0001] 1. Field Of The Invention

[0002] The present invention relates to methods of motion estimation ina sequence of moving video pictures. More specifically, but notexclusively, the invention relates to methods of motion vectorestimation and an apparatus for the same.

[0003] 2. Description of the Prior Art

[0004] A prior art motion estimation technique, called “3-D RecursiveSearch,” has been described by Gerard de Haan and P. W. A. C. Biezen, in“Sub-pixel motion estimation with 3-D recursive search block-matching,”Signal Processing: Image Communication 6, pp. 229-239 (1994),incorporated herein by reference as if set forth in full.

[0005] 3-D Recursive Search falls in the class of block-recursive motionestimators. The algorithm is based on the assumptions that motion doesnot change much in time, i.e., from field to field. The algorithmmaintains a vector field which is updated on field basis. The vectorfield is usually similar for a relatively large region, i.e., for anobject. Therefore the motion vectors in the neighborhood of a locationare good candidates for the motion at that location.

[0006] A motion video consists of a sequence of fields. Each field isdivided into blocks, e.g., of 16 by 16 pixels. A motion vector isassociated with each block. The motion vector should hold thedisplacement between the block in the current frame compared to aprevious field or next field. For example, to update the motion vectorof block (x, y) in a current field, a 3-D Recursive Search uses only alimited number of candidate vectors, say five, for the estimation,namely, some vectors from the previous field, i.e., temporal vectors,some vectors from the current field, i.e., spatial vectors, and anupdate of a spatial vector. For each candidate the motion estimationerror is calculated. The candidate with the lowest motion estimationerror is chosen as the best motion vector for that block. The algorithmuses the normal raster scan order to go through the blocks.

[0007] 3-D recursive search estimators are also described by G. de Haan,et al. in “True Motion Estimation with 3-D Recursive search blockmatching. IEEE Trans. Circuits and Systems for Video Technology, Vol. 3,October 1993, pp. 368-379; and in “Sub-pixel motion estimation with 3-Drecursive search block matching; Signal Processing: Image Communications6, pp. 229-239 (1994), incorporated herein by reference as set forth infull. Unlike the more expensive full-search block matchers that estimateall the possible displacements within a search area, this algorithm onlyinvestigates a very limited number of possible motion vectors. Bycarefully choosing the candidate vectors, a good performance can beachieved, approaching almost true motion, with a low complexity design.

[0008] Motion estimation is useful in several applications. It is partof predictive coding applications like MPEG-2, H.263, and the like. Assuch, the motion vectors are being used to maximize temporalcorrelation, and therefore, to minimize the coding error. Motionestimation is also being applied in the field of video enhancement, forexample, to improve the motion portrayal of movie pictures,deinterlacing, or temporal noise reduction.

[0009] Motion can be estimated in several ways. For example, motionestimators include a full-search estimator, block-matching, object-basedmethods, and the like. Nevertheless, they all try to maximize temporalcorrelation by assuming a certain spatial-invariant motion model. As anexample, the 3-D-Recursive Search motion estimator, used in the PhillipsNatural Motion TV sets, estimates translational motion on a block basis.

[0010] In the 3-D Recursive Search Estimator, the candidate motionvectors are selected on a per block basis. An error function, e.g., amean squared error or mean absolute difference, is calculated percandidate. A penalty is added which depends on the candidate type, e.g.,spatial or temporal. The penalty per candidate type is spatiallyinvariant, i.e., it does not vary with the spatial position of theblock. As a result, the use of this type of penalty only may result insuboptimal coding gain and some artifacts in the picture.

[0011] It is, therefore, an object of the present invention to providean improved motion estimation method. It is another object of theinvention to increase the speed of convergence of motion vectors toimprove the convergence process.

SUMMARY OF THE INVENTION

[0012] The present invention, which addresses the needs of the priorart, provides an improved motion vector estimation method and a devicefor the same. The method of the invention includes providing a motionestimator, obtaining at least two candidate motion vectors from themotion estimator, and applying an error function having a penalty thatdepends on the position and size of the candidate motion vectors. Eachcandidate motion vector is associated with a region representation of avideo image.

[0013] The present invention is also a device for recursive motionvector estimation having enhanced convergence including a random vectorgenerator for generating a plurality of candidate motion vectorsassociated with selected regions in at least a first and a second videoimage, a best vector selector for comparing the candidate motion vectorsof selected regions in a first and a second video image, the best vectorselector including means for evaluating the candidate motion vectors byapplying an error function having at least a penalty that depends on theposition and size of the candidate motion vectors.

[0014] Other improvements which the present invention provides over theprior art will be identified as a result of the following descriptionwhich sets forth the preferred embodiments of the present invention. Thedescription is not in any way intended to limit the scope of the presentinvention, but rather only to provide the working example of the presentpreferred embodiments. The scope of the present invention will bepointed out in the appended claims.

DETAILED DESCRIPTION OF THE INVENTION

[0015] The method according to the invention, which is described below,includes providing a motion estimator which can select at least twocandidate-motion vectors and applying to the selected candidate motionvectors an error function containing a spatial penalty that depends bothon the position and size of the selected candidate.

[0016] A motion estimation device includes means to compare pictureblocks of one picture with multiple picture blocks in another picture.The evaluation is for a block matching algorithm on the basis of regionsor blocks, typically but not limited to 8 by 8 pixels or 16 by 16.Several blocks of another picture are compared with a block in thecurrent picture for which a motion vector is being searched by comparingthe pixel value contained in these blocks. The evaluation isaccomplished by using an error function, for example a sum-of-absolutedifferences (SAD), or a mean-squared-error (MSE). In the SAD method allor some of the pixel values within the corresponding blocks aresubtracted from each other and evaluated in the error function. Usually,the vector yielding the smallest error is selected as the best matchproviding the best vector. The corresponding candidate motion vector isselected as the motion vector for the current block. The process startsagain for the next block, until all blocks with the current picture aredetermined.

[0017] “A best vector selector”, as used herein, refers to a device ormethod that selects the nearest matching candidate vector.

[0018] Useful motion estimators for the present invention include,without limitation, block matching and recursive-pel types as moreparticularly described by G. de Haan in “Progress in Motion Estimationfor Consumer Video Format Conversation”, IEEE Transactions on ConsumerElectronics, vol. 46, no. 3, pp. 449-459, (August 2000) incorporatedherein by reference as if set forth in full. A preferred block matchingestimator useful in the invention is the 3-Dimensional Recursive SearchBlock Estimator, as described in U.S. Pat. Nos. 5,212,548; 5,072,293;5,148,269, all to de Haan et al., incorporated herein by reference asset forth in full. In addition, useful motion estimators for the presentinvention, include a motion estimation device as described above whichcan operate not only on blocks but regions of any kind of shapeincluding blocks and pixels.

[0019] In selecting the candidate motion vectors, a video picture, F({right arrow over (x)}, n) was considered, where$\overset{\rightarrow}{x} = \begin{pmatrix}x \\y\end{pmatrix}$

[0020] defined the spatial position and n the field number, i.e., thetemporal position, and F( ) was the pixel value at the spatio-temporalposition defined by {right arrow over (x)} and n . The motion, {rightarrow over (d)}({right arrow over (x)}, n), for every pixel in thepicture was estimated. However, since the motion estimation process wascomputation intensive and improvement in consistency with existingmotion estimators was sought, the estimation process was limited toblocks of pixels, typically 8 by 8 pixels. As such, the motion, {rightarrow over (d)}({right arrow over (x)}, n) was estimated for every 8 by8 block in the picture wherein {right arrow over (X)} represents thecenter of a block.

[0021] Again, in order to reduce computational complexity and improvespatio-temporal consistency, only a few, typical four to five, candidatemotion vectors ({right arrow over (C)}) per block were evaluated asdescribed in by G. de Haan and P. W. A. C. Bizen in “Sub-pixel motionestimation with 3-D recursive search block matching,” Signal Processing:Image Communication, vol. 6, pp. 229-239 (1994). As used herein“spatio-temporal consistency” refers to a consistency in both space andtime. The evaluation was based on calculating an error function,typically a sum of absolute differences, though not limited to this:

ε({right arrow over (C)},{right arrow over(X)},n)=Σ_({right arrow over (x)}εB({right arrow over (X)},n)) |F({rightarrow over (x)},n)−F({right arrow over (x)}−{right arrow over(C)}({right arrow over (x+EE,n),n−1)|(1) )}

[0022] where B({right arrow over (X)},n) is the block of pixels with thecenter {right arrow over (X)}.

[0023] The candidates were selected from a spatio-temporal neighborhoodas more particularly described by G. de Haan and P. W. A. C. Bizen in“Sub-pixel motion estimation with 3-D recursive search block matching,”Signal Processing: Image Communication, vol. 6, pp. 229-239 (1994). Thebest match, i.e., the candidate yielding the smallest cost or error, isselected as the motion vector d({right arrow over (X)},n) for theevaluated block.

[0024] To further improve the consistency, a penalty, P, was added whichdepends on the ‘type’ of the candidate motion vector. As such, spatialcandidates, i.e., candidate motion vectors that are already calculatedin the current picture, could be given a different penalty then temporalcandidates, i.e., candidate motion vectors that are known from thecalculation in the previous picture(s):

ε({right arrow over (C)},{right arrow over(X)},n)=Σ_({right arrow over (x)}εB({right arrow over (X)},n)) |F({rightarrow over (x)},n)−F({right arrow over (x)}−{right arrow over(C)}({right arrow over (x+EE,n), n−1)|+P(C)})   (2)

[0025] It has been found that the statistics of motion vectorsassociated with a video region reveal that the probability of velocitiesin the picture depend on the position of the screen. It is, therefore,useful to extend the model described above for motion estimation. Byadding an additional penalty that depends on the spatial position andsize, the motion estimator can be biased towards the found statistics:

ε({right arrow over (C)},{right arrow over(X)},n)=Σ_({right arrow over (x)}εB({right arrow over (X)},n)) |F({rightarrow over (x)},n)−F({right arrow over (x)}−{right arrow over(C)}({right arrow over (x+EE,n), n−1)|+P(C)})+P(∥{right arrow over(C)}∥,{right arrow over (X)})   (3)

[0026] where ∥{right arrow over (C)}∥ is the norm (ie., the ‘size’) ofthe candidate motion vector. As an example, the penalty can berelatively large for large candidate motion vectors in picture blocks{right arrow over (X)} towards the outside of the picture, and also forsmall candidate motion vectors in picture blocks evaluated for thecenter part of the picture.

[0027] The term P(∥{right arrow over (C)}∥,{right arrow over (X)}) isnot limited to be used by the 3D-recursive search motion estimator only.Any motion estimator needs to evaluate candidates, which can even be avery large set, according to a certain cost function. The addition ofthe candidate size and position dependent term in the cost functionallow biasing the motion estimator, and is therefore, the preferredembodiment of the present invention.

[0028] This relative simple addition to the cost function allows afaster convergence of the motion vector towards the ‘real’ one. In thismanner a dependency was found that helped to improve the convergence. Itis noted that based on evaluating only a few candidates, it might takeseveral iterations or passes, most often in the temporal direction,before the real motion vector is found, and convergence is established.Moreover, a faster convergence directly implies an improved overallaverage accuracy of the motion vectors.

[0029] Thus, while we have described what are the preferred embodimentsof the present invention, further changes and modifications can be madeby those skilled in the art without departing from the true spirit ofthe invention, and it is intended to include all such changes andmodifications as come within the scope of the claims set forth below.

1. A method to enhance motion estimation, said method comprising:providing a motion estimator; obtaining at least two candidate motionvectors from said motion estimator; and applying an error functionhaving a penalty that depends on the position and size of said candidatemotion vectors.
 2. The method of claim 1, wherein said candidate motionvectors are associated with a region representation of a video image. 3.A device for recursive motion vector estimation having enhancedconvergence which comprises: a vector generator for generating aplurality of candidate motion vectors associated with selected regionsin at least a first and a second video image; a best vector selector forcomparing said candidate motion vectors of selected regions in a firstand a second video image, said best vector selector comprising means forevaluating said candidate motion vectors by applying an error functionhaving at least a penalty that depends on the position and size of saidcandidate motion vectors.
 4. The method of claim 3, wherein the errorfunction is calculated by a function of the sum-of-the-absolutedifferences or the mean-squared error.